Advances in Multimedia Modeling
Abstract. Attention is a psychological measurement of human reflection against stimulus. We propose a general framework of highlight detection by comparing attention intensity during the watching of sports videos. Three steps are involved: adaptive selection on salient features, unified attention estimation and highlight identification. Adaptive selection computes feature correlation to decide an optimal set of salient features. Unified estimation combines these features by the technique of multi-resolution auto-regressive (MAR) and thus creates a temporal curve of attention intensity. We rank the intensity of attention to discriminate boundaries of highlights. Such a framework alleviates semantic uncertainty around sport highlights and leads to an efficient and effective highlight detection. The advantages are as follows: (1) the capability of using data at coarse temporal resolutions; (2) the robustness against noise caused by modality asynchronism, perception uncertainty and feature mismatch; (3) the employment of Markovian constrains on content presentation, and (4) multi-resolution estimation on attention intensity, which enables the precise allocation of event boundaries. Key words:highlight detection, attention computation, sports video analysis 1
Nearest%2Dneighbor classification using unlabeled data for real world image application.

Large scale partially duplicated web image retrieval.

Multi%2Dlabel boosting for image annotation by structural grouping sparsity.

Large scale partially duplicated web image retrieval.

S3MKL scalable semi%2Dsupervised multiple kernel learning for image data mining.

The use of non%2Dconventional methods for content analysis and understanding panel overview.

Learning to photograph.
this paper we present a design framework for the creation of learning activities that draw on interest as their primary source of motivation. of learning activities that draw on, framework is the embodiment of a theory that addresses two challenges to the large-scale implementation of interest-driven learning. addresses two challen  coverage, the di iculty o motivating the wide range o learning objectives valued by our educational system with interest, and stakaI)l  the variation over time and across learners o the motivational strength o  interest in a particular topic or activity. o,RzG:A,MHbbjj,.G:,MHRTjj,. ramework is based on a novel categorization o motivational constructs into interest and context-based motivators. MOTIVATING ACTIVE LEARNING: A DESIGN FRAMEWORK FOR INTERESTDRIVEN LEARNING Common sense and cognitive theory concur: Interest and engagement lead to robust learning. Translating that recognition into clear principles for learning environment design is a complexproblem. How can materials, teachers, peer groups, and activity structures be designed to capitalize on interest as a motivation to learn? This question exposes a critical gap between the literature on the design of constructivist learning environments and the educational psychology literature onmotivation. Recent research on constructivist learning environment design has argued for the motivational importance of authentic, interesting tasks and contexts(e.ex Blumenfeld, 1992; Cognition and Technology Group at Vanderbilt, 1992; Guthrie &amp; Alao, 1997; Schank, Fano, Bell, &amp; Jona,1993/1994). The motivation research literature describes the value of the same kind of motivation in studies and in theoretical work regardinginterest. However, neither literature specifically addresses the challenges of des...
Building contextual visual vocabulary for large%2Dscale image applications.

Spatial coding for large scale partial%2Dduplicate web image search.

Real%2Dworld trajectory extraction for attack pattern analysis in soccer video.

Latent visual context analysis for image re%2Dranking.

Music video affective understanding using feature importance analysis.

Cross%2Ddatabase age estimation based on transfer learning.

Multi%2Dlevel trajectory modeling for video copy detection.

Large scale partial%2Dduplicate image retrieval with bi%2Dspace quantization and geometric consistency.

Large scale partial%2Dduplicate image retrieval with bi%2Dspace quantization and geometric consistency.

Building pair%2Dwise visual word tree for efficent image re%2Dranking.

Fast traumatic brain injury CT slice indexing via anatomical feature classification.
Computed tomography (CT) is used widely in traumatic brain injury diagnosis. One axial brain CT scan consists of multiple slices with different heights along the brain axial direction. Indexing of brain CT slices is to order the slices and align each individual slice onto the corresponding brain axial height, which is an important step in content-based image retrieval and computer-assisted diagnosis. Current existing methods for this indexing task are through the image registration techniques by registering 2D image slices onto a 3D brain atlas. In this paper, instead of using the registration methods, we propose a fast indexing method using anatomical feature classification. In our method, the brain CT scan is divided into 6 height levels along the axial direction so that slices in each level share similar anatomical structure. In this way, the indexing problem becomes a classification problem that one series of scan slices are to be classified into 6 classes. Experimental results show that the proposed method is effective and efficient.
Multiple instance learning using visual phrases for object classification.

Action Recognition Using Spatial%2DTemporal Context.

Multiple Kernel Learning with High Order Kernels.

Automatic Pathology Annotation on Medical Images A Statistical Machine Translation Framework.

Group Activity Recognition by Gaussian Processes Estimation.

Interactive Web Video Advertising with Context Analysis and Search.

Canonical Image Selection by Visual Context Learning.

AdVR Linking Ad Video with Products or Service.

Correlation%2DBased Feature Selection and Regression.

Image Histogram Constrained SIFT Matching.

The 1st workshop on large%2Dscale multimedia retrieval and mining %28LS%2DMMRM%2709%29.

Descriptive visual words and visual phrases for image applications.

Multimedia content analysis model%2Dbased approaches vs. data%2Ddriven approaches.

Construction of a linear unbiased diffeomorphic probabilistic liver atlas from CT images.

Utilizing affective analysis for efficient movie browsing.

From hemorrhage to midline shift A new method of tracing the deformed midline in traumatic brain injury ct images.
In intracranial pathological examinations using CT scan, brain midline shift (MLS) is an important diagnostic feature indicating the pathological severity and patient’s survival possibility. In this paper, we develop a new method of tracing the brain midline shift in traumatic brain injury (TBI) CT images using its original cause- the hemorrhage. Firstly, we model the relationship between the hemorrhage and the midline deformation caused by it using a linear regression model (H-MLS model). Secondly, using the H-MLS model, the deformed midline is predicted from the hemorrhage detected in CT images. Finally, the predicted deformed midline is adjusted according to the visual symmetry information. Preliminary experiments show that the proposed method is effective and time-efficient. Index Terms — Medical image analysis, midline shift, computed tomography, traumatic brain injury, hemorrhage 1.
Category sensitive codebook construction for object category recognition.

What can visual content analysis do for text based image search%3F

A lexica family with small semantic gap.
Defining a lexicon of high-level concepts is the first step for data collection and model construction in concept-based image retrieval. Differences of semantic gaps among concepts are well worth considering. By measuring consistency in visual space and textual space, concepts with small semantic gap can be obtained. Considering so many diverse concepts in large-scale image dataset, we construct a lexica family of high-level concepts with small semantic gap based on different low-level features and different consistency measurements. In this lexica family, the lexica are independent to each other and mutually complementary. It provides helpful suggestions about data collection, feature selection and search model construction for large-scale image retrieval. Index Terms — semantic gap, lexica, large-scale, image retrieval
Refining image retrieval using one%2Dclass classification.

Visual event detection using orientation histograms with feature point trajectory information.

Visual event detection using orientation histograms with feature point trajectory information.

Toward a higher%2Dlevel visual representation for object%2Dbased image retrieval.

Probabilistic optimized ranking for multimedia semantic concept detection via RVM.

i.MTV an integrated system for mtv affective analysis.

Probabilistic optimized ranking for multimedia semantic concept detection via RVM.

Visual Synset Towards a higher%2Dlevel visual representation.

What are the high%2Dlevel concepts with small semantic gaps%3F
Concept-based multimedia search has become more and more popular in Multimedia Information Retrieval (MIR). However, which semantic concepts should be used for data collection and model construction is still an open question. Currently, there is very little research found on automatically choosing multimedia concepts with small semantic gaps. In this paper, we propose a novel framework to develop a lexicon of high-level concepts with small semantic gaps (LCSS) from a large-scale web image dataset. By defining a confidence map and content-context similarity matrix, images with small semantic gaps are selected and clustered. The final concept lexicon is mined from the surrounding descriptions (titles, categories and comments) of these images. This lexicon offers a set of high-level concepts with small semantic gaps, which is very helpful for people to focus for data collection, annotation and modeling. It also shows a promising application potential for image annotation refinement and rejection. The experimental results demonstrate the validity of the developed concepts lexicon. 1.
Text Mining in Radiology Reports.
Medical text mining has gained increasing interest in recent years. Radiology reports contain rich information describing radiologist’s observations on the patient’s medical conditions in the associated medical images. However, as most reports are in free text format, the valuable information contained in those reports cannot be easily accessed and used, unless proper text mining has been applied. In this paper, we propose a text mining system to extract and use the information in radiology reports. The system consists of three main modules: a medical finding extractor, a report and image retriever, and a text-assisted image feature extractor. In evaluation, the overall precision and recall for medical finding extraction are 95.5 % and 87.9% respectively, and for all modifiers of the medical findings 88.2 % and 82.8 % respectively. The overall result of report and image retrieval module and text-assisted image feature extraction module is satisfactory to radiologists. 1.
Affective MTV analysis based on arousal and valence features.

Learning object from small and imbalanced dataset with Boost%2DBFKO.

Hemorrhage slices detection in brain CT images.

Object%2DBased Image Retrieval Beyond Visual Appearances.

Personalized MTV Affective Analysis Using User Profile.

Similarity Matching in Computer Vision and Multimedia.
Comparing two images, or an image and a model, is the fundamental operation for any retrieval systems. The similarity matching of two images can reside in the hierarchical levels from pixel-by-pixel level, feature space level, object level, and semantic level. In most systems of interest, a simple pixel-by-pixel comparison will not suffice: the difference that we determine must bear some correlation with the perceptual difference of the two images or with the difference between two adequate semantics associated to the two images. Similarity matching techniques are developed mostly for recognition of objects under several conditions of the distortion while similarity measures, on the other hand, are used in applications like image databases. Matching and dissimilarity measurement are not seldom based on the same techniques, but they differ in emphasis and applications.
Semantic Subspace Projection and Its Applications in Image Retrieval.

Distance Learning for Similarity Estimation.
Abstract—In this paper, we present a general guideline to find a better distance measure for similarity estimation based on statistical analysis of distribution models and distance functions. A new set of distance measures are derived from the harmonic distance, the geometric distance, and their generalized variants according to the Maximum Likelihood theory. These measures can provide a more accurate feature model than the classical euclidean and Manhattan distances. We also find that the feature elements are often from heterogeneous sources that may have different influence on similarity estimation. Therefore, the assumption of single isotropic distribution model is often inappropriate. To alleviate this problem, we use a boosted distance measure framework that finds multiple distance measures, which fit the distribution of selected feature elements best for accurate similarity estimation. The new distance measures for similarity estimation are tested on two applications: stereo matching and motion tracking in video sequences. The performance of boosted distance measure is further evaluated on several benchmark data sets from the UCI repository and two image retrieval applications. In all the experiments, robust results are obtained based on the proposed methods.
An Efficient Sequential Approach to Tracking Multiple Objects Through Crowds for Real%2DTime Intelligent CCTV Systems.

Automatic composition of broadcast sports video.

Adaptive discriminant analysis for microarray%2Dbased classification.

Feature selection using principal feature analysis.
Dimensionality reduction of a feature set is a common preprocessing step used for  pattern recognition and classification applications and in compression schemes.  Principal component analysis (PCA) is one of the popular methods used, and can be  shown to be optimal using different optimality criteria. However, it has the  disadvantage that measurements from all of the original features are used in the  projection to the lower dimensional space. This paper proposes a novel method for  dimensionality reduction of a feature set by choosing a subset of the original features  that contains most of the essential information, using the same criteria as the PCA.  We call this method Principal Feature Analysis (PFA). The proposed method is  successfully applied for choosing the principal features in face tracking and contentbased  image retrieval problems.  1 
The use of temporal
We give an efficient procedure for verifying that a finite-state concurrent system meets a specification expressed in a (propositional, branching-time) temporal logic. Our algorithm has complexity linear in both the size of the specification and the size of the global state graph for the concurrent system. We also show how this approach can be adapted to handle fairness. We argue that our technique can provide a practical alternative to manual proof construction or use of a mechanical theorem prover for verifying many finite-state concurrent systems. Experimental results show that state machines with several hundred states can be checked in a matter of seconds.
Automatically Discovering Unknown Short Video Repeats.

Integrating Relevance Feedback in Boosting for Content%2DBased Image Retrieval.

Outlier Detection from Pooled Data for Image Retrieval System Evaluation.

Two%2DDimensional Adaptive Discriminant Analysis.

Semantic Analysis and Personalization for Mobile Media Applications.

i.Boosting for Image Classification.

Interactive Boosting for Image Classification.

Interactive Boosting for Image Classification.

Personalized multimedia retrieval the new trend%3F
The aim of this paper is to present an overview of the current situation in this hot topic of Multimedia Information Retrieval: Personalization. We are considering several aspects of this problem. On one hand, the user will want to have a personalized access to his image/video collections and this can be achieved by providing intuitive and natural browsing capabilities and customized features. Furthermore, the system is required to perform user profiling and to adapt the existing parameters of the system to the user needs and interest. On the other hand, it is also important to consider the devices and applications in which this technology is going to be deployed. Mobile media is a high growth area but the state-of-the-art technologies are lagging behind the consumers expectations. We are addressing in this paper precisely these three important aspects.
FADA An Efficient Dimension Reduction Scheme for Image Classification.

Classification of CT Brain Images of Head Trauma.
Abstract. A method for automatic classification of computed tomography (CT) brain images of different head trauma types is presented in this paper. The method has three major steps: 1. The images are first segmented to find potential hemorrhage regions using ellipse fitting, background removal and wavelet decomposition technique; 2. For each region, features (such as area, major axis length, etc.) are extracted; 3. Each extracted feature is classified using machine learning algorithm; the images are then classified based on its component regions’ classification. The automatic medical image classification will be useful in building a content-based medical image retrieval system. 1
Learning Microarray Gene Expression Data by Hybrid Discriminant Analysis.

Integrating Discriminant and Descriptive Information for Dimension Reduction and Classification.

Generation of Personalized Music Sports Video Using Multimodal Cues.

Efficient Short Video Repeat Identification With Application to News Video Structure Analysis.

Interactive Semisupervised Learning for Microarray Analysis.

An algorithm to estimate mean vehicle speed from MPEG Skycam video.

Content%2Dadaptive digital music watermarking based on music structure analysis.

News video search with fuzzy event clustering using high%2Dlevel features.
Precise automated video search is gaining in importance as the amount of multimedia information is increasing at exponential rates. One of the drawbacks that make video retrieval difficult is the lack of available semantics. In this paper, we propose to supplement the semantic knowledge for retrieval by providing useful semantic clusters derived from event entities present in the news video. These entities include the output from keywords derived from the automated speech recognition (ASR) and eventrelated High-level Features (HLF) extracted from the news video at the pseudo story level. Fuzzy clustering is then carried out to group similar stories together to form semantic clusters. The retrieval system utilizes these clusters to refine the re-ranking process in the Pseudo Relevance Feedback (PRF) step. Initial experiments performed on video search task using the TRECVID 2005 dataset show that the proposed approach can improve the search performance significantly.
Learning image manifolds by semantic subspace projection.

Stripe Image Feature Based on a New Grid Method and Its Application in ImageCLEF.

Toward Robust Distance Metric Analysis for Similarity Estimation.
In this paper, we present a general guideline to establish the relation between a distribution model and its corresponding similarity estimation. A rich set of distance metrics, such as harmonic distance and geometric distance, is derived according to Maximum Likelihood theory. These metrics can provide a more accurate feature model than the conventional Euclidean distance (SSD) and Manhattan distance (SAD). Because the feature elements are from heterogeneous sources and may have different influence on similarity estimation, the assumption of single isotropic distribution model is often inappropriate. We propose a novel boosted distance metric that not only finds the best distance metric that fits the distribution of the underlying elements but also selects the most important feature elements with respect to similarity. We experiment with different distance metrics for similarity estimation and compute
An Automatic Classification System Applied in Medical Images.

Toward Intelligent Use of Semantic Information on Subspace Discovery for Image Retrieval.

TV Commercial Classification by using Multi%2DModal Textual Information.

A New Study on Distance Metrics as Similarity Measurement.
Distance metric is widely used in similarity estimation. In this paper we find that the most popular Euclidean and Manhattan distance may not be suitable for all data distribution. A general guideline to establish the relation between a distribution model and its corresponding similarity estimation is proposed. Based on Maximum Likelihood theory, we propose new distance metrics, such as harmonic distance and geometric distance. Because the feature elements may be from heterogeneous sources and usually have different influence on similarity estimation, it is inappropriate to model the distribution as isotropic. We propose a novel boosted distance metric that not only finds the best distance metric that fits the distribution of the underlying elements but also selects the most important feature elements with respect to similarity. The boosted distance metric is tested on fifteen benchmark data sets from the UCI repository and two image retrieval applications. In all the experiments, robust results are obtained based on the proposed methods. 1.
Local Motion Analysis and Its Application in Video based Swimming Style Recognition.


Efficient Relevance Feedback Using Semi%2Dsupervised Kernel%2Dspecified K%2Dmeans Clustering.


A Self%2Dsupervised Learning Framework for Classifying Microarray Gene Expression Data.

A Foundational Perspective for Visual Information Retrieval.

A Secure and Robust Authentication Scheme for Video Transcoding.

Nonparametric motion characterization for robust classification of camera motion patterns.

Trajectory%2DBased Ball Detection and Tracking in Broadcast Soccer Video.

Automatic summarization of music videos.

A novel multi%2Dresolution video representation scheme based on kernel PCA.

Proceedings of the 7th ACM SIGMM International Workshop on Multimedia Information Retrieval

MultiPRE a novel framework with multiple parallel retrieval engines for content%2Dbased image retrieval.
We propose a novel framework for content-based image retrieval with multiple parallel retrieval engines (MultiPRE) to achieve higher retrieval performance. Visual features, including both low-level features, such as color, texture and region features, and middle-level structure features, such as blob representation of objects are used to capture geometrical and statistical characteristics of images. Both clustering analysis and discrimination analysis are used as similarity measures in multiple retrieval engines, which are based on principal component analysis (PCA) and support vector machines (SVM), respectively. Finally outputs of these engines are fused to determine ranking lists of retrieved images for given retrieval topics. The proposed framework has been evaluated based on the 26 image query topics over the CasImage database with over 9000 medical images used in ImageCLEF 2004, an international research effort for content-based image retrieval performance benchmark. Experiments show that the proposed framework achieved significantly better performance in terms of both the mean and the variance of average precision than the best run reported in ImageCLEF2004.
A repeated video clip identification system.

Automatic generation of personalized music sports video.
In this paper, we propose a novel automatic approach for personalized music sports video generation. Two research challenges, semantic sports video content selection and automatic video composition, are addressed. For the first challenge, we propose to use multi-modal (audio, video and text) feature analysis and alignment to detect the semantic of events in sports video. For the second challenge, we propose video-centric and music-centric music video composition schemes to automatically generate personalized music sports video based on user’s preference. The experimental results and user evaluations are promising and show that our system’s generated music sports video is comparable to manually generated ones. The proposed approach greatly facilitates the automatic music sports video generation for both professionals and amateurs.
Hybrid PCA and LDA Analysis of Microarray Gene Expression Data.

Combining Visual Features for Medical Image Retrieval and Annotation.

Content%2Dbased medical image retrieval using dynamically optimized regional features.

Replay Scene Classification in Soccer Video Using Web Broadcast Text.

Neighborhood issue in single%2Dframe image super%2Dresolution.

Video Object Boundary Reconstruction by 2%2DPass Voting.

A Mid%2Dlevel Visual Concept Generation Framework for Sports Analysis.

Periodicity Detection of Local Motion.

A unified framework for semantic shot representation of sports video.

Boosting Multiple Classifiers Constructed by Hybrid Discriminant Analysis.

A unified framework for semantic shot classification in sports video.
In this demonstration, we present a unified framework for semantic shot classification in sports videos. Unlike previous approaches, which focus on clustering by aggregating shots with similar low-level features, the proposed scheme makes use of domain knowledge of specific sport to perform a top-down video shot classification. That is, combining with inherent game rules and television field production, for each sport through careful observations we predefine a set of semantic shots which cover 90 to 95 % of sports broadcasting video. Under the supervision of predefined shots set, we map the low-level features to high-level semantic video shot attributes such as dominant object motion (a player), persistent camera panning, and court shape. On the basis of the appropriate fusion of those high-level shot attributes, we classify video shots into several predefined categories, each of which has a clear semantic meaning. The experiments show that, compared to traditional clustering methods and key-frame based analysis, the proposed framework features great capability of semantics mining. Due to remarkable structure constraints and limited sports photography, this framework provides a generic solution for sports video shot classification, which can be adapted to a new sport type without major modification. With correctly classified sports video shots further structural and temporal analysis will be greatly facilitated.
Self%2Dsupervised learning based on discriminative nonlinear features for image classification.

A robust and accumulator%2Dfree ellipse hough transform.
The ellipse Hough transform (EHT) is a widely-used technique. Most of the previous modifications to the standard EHT improved either the voting procedure that computes the absolute measure function (AMF) or the peak detection of the AMF. However, existing EHTs are not robust for detecting partial slightly-oblique ellipses. This paper presents a Robust and Accumulator-Free Ellipse Hough Transform (RAF-EHT), an improved EHT that is robust even for partial slightly-oblique ellipses. Our RAF-EHT is based on two main ideas, namely, (1) an improved measure function (IMF) for handling the partiality and the obliqueness of ellipses, (2) a new accumulator-free computation scheme for finding the top k peaks of the IMF, without complex peak detection. Experimental results show that the RAF-EHT is more robust than the existing EHTs in detecting the partial slightlyoblique ellipses. In addition, the RAF-EHT needs only a little memory because it is accumulator-free.
A color fingerprint of video shot for content identification.

Automatic replay generation for soccer video broadcasting.

Nonparametric motion model with applications to camera motion pattern classification.

Nonparametric motion model.
The problem to segment an image sequence in terms of regions characterized by a coherent motion is among the most challenging in image sequence analysis. This paper proposes a new technique which sequentially renes the segmentation and the motion estimation by combining static segmentation and motion information. Simulation results show the efficiency of the proposed technique.  
Fast and robust video clip search using index structure.

Mean shift based nonparametric motion characterization.

Goal detection in soccer video using audio%2Fvisual keywords.

Event detection based on non%2Dbroadcast sports video.
Most recent research work on sports video analysis focuses on broadcast video. The broadcast video is a post-produced video and has much additional editing information inserted. In this paper, we propose a novel event detection framework in sports game only based on the video taken by single (main) camera. Compared with event detection from broadcast video, the proposed framework is more challenging as interesting events needs to be automatically detected. The results of the research could be used for automatic replay generation in sports video. In this paper, a mid-level representation is first created based on low-level audio/visual feature and events related to the replay are then detected from this representation. We experiment our framework and some promising results are obtained. 1.
Toward an improved error metric.
In many computer vision algorithms, the well known Euclidean or SSD (sum of the squared differences) metric is prevalent and justified from a maximum likelihood perspective when the additive noise is Gaussian. However, Gaussian noise distribution assumption is often invalid. Previous research has found that other metrics such as double exponential metric or Cauchy metric provide better results, in accordance with the maximum likelihood approach. In this paper, we examine different error metrics and provide a theoretical approach to derive a rich set of nonlinear estimations. Our results on image databases show more robust results are obtained for noise estimation based on the proposed error metric analysis. 1.
News sports video shot classification with sports play field and motion features.

A generic mid%2Dlevel representation for semantic video analysis.

A new analysis of the value of unlabeled data in semi%2Dsupervised learning for image retrieval.
There has been an increasing interest in using unlabeled data in semi-supervised learning for various classification problems. Previous work shows that unlabeled data can improve or degrade the classification performance depending on whether the model assumption matches the ground-truth data distribution, and also on the complexity of the classifier compared with the size of the labeled training set. In this paper, we provide a new analysis on the value of unlabeled data by considering different distributions of the labeled and unlabeled data and showing the migrating effect for semi-supervised learning. Extensive experiments have been performed in the context of image retrieval application. Our approach evaluates the value of unlabeled data from a new aspect and is aimed to provide a guideline on how unlabeled data should be used. 1.
A hierarchical approach to story segmentation of large broadcast news video corpus.

A robust Hough%2Dbased algorithm for partial ellipse detection in broadcast soccer video.

Automatically summarize musical audio using adaptive clustering.
Automatic music summarization is very useful for music indexing, content-based music retrieval and on-line music distribution, but it is a challenge to automatically extract the most common and salient themes from unstructured music raw data. In this paper, we propose an effective approach to automatically summarize music content. Firstly, a number of features are extracted to characterize the music content. Based on extracted features, an adaptive clustering algorithm is then applied to structure the music content. Finally, the music summary is created in terms of the clustering results and domain-related music knowledge. User study is conducted to evaluate the quality of summarization. The experiments on different genres of music illustrate the results of summarization are significant and effective to actual expectation. 1.
Parameterized discriminant analysis for image classification.

Two dimensional timeline and its application to Conversant Media system.

A statistical approach for object motion estimation with MPEG motion vectors.

Sports highlight detection from keyword sequences using HMM.
Sports video highlight detection is a popular topic. A multi-layer sport event detection framework is described in this paper. In the mid-level of this framework visual and audio keywords are created from low-level features and the original video is converted into keyword sequence. In the high-level the temporal pattern of keyword sequences are analyzed by HMM classifier. The creation of visual and audio keyword can help to bridge the gap between low-level features and high-level semantic. The use of HMM classifier can automatically find the temporal change character of the event instead of rule based heuristically modeling to map certain keyword sequence into events. We experiment our model on soccer games and some promising results are obtained. 1.
Robust multi%2Dlevel video representation using mean shift analysis.

Indexing with Musical Events and Its Application to Content%2DBased Music Identification.

Visual Keywords Labeling in Soccer Video.

Fast and Robust Search Method for Short Video Clips from Large Video Collection.

Robust Error Metric Analysis for Noise Estimation in Image Indexing.

Multi%2DLevel Video Representation with Application to Keyframe Extraction.

Fast and robust short video clip search using an index structure.
In this paper, we present an index structure-based method to fast and robustly search short video clips in large video collections. First we temporally segment a given long video stream into overlapped matching windows, then map extracted features from the windows into points in a high dimensional feature space, and construct index structures for these feature points for querying process. Different from linear-scan similarity matching methods, querying process can be accelerated by spatial pruning brought by an index structure. A multi-resolution kd-tree (mrkd-tree) is employed to complete exact K-NN Query and range query with the aim of fast and precisely searching out all short video segments having the same contents as the query. In terms of feature representation, rather than selecting representative key frames, we develop a set of spatial-temporal features in order to globally capture the pattern of a short video clip (e.g. a commercial clip, a lead in/out clip) and combine it with the color range feature to form video signatures. Our experiments have shown the efficiency and effectiveness of the proposed method that the very first instance of a given 10-sec query clip can be identified from a 10.5hour video collection in tens of milliseconds. The proposed method has been also compared with the fast sequential search algorithm.
Automatic Sports Highlights Extraction with Content Augmentation.
Abstract. We describe novel methods to automatically augment content into video highlights detected from soccer and tennis video. First, we extract generic and domain-specific features from the video to isolate key audio-visual events that we have empirically found to correlate well with the ground-truth highlights. Next, based on a set of heuristics-driven rules to minimize view disruption, spatial regions in the image frames of these video highlight segments are segmented for content augmentation. Preliminary trials from subjective viewing indicate a high level of acceptance for the content insertions. 1
Complete Performance Graphs in Probabilistic Information Retrieval.

Implanting Virtual Advertisement into Broadcast Soccer Video.

Archiving Tennis Video Clips Based on Tactics Information.

Fast and Robust Short Video Clip Search for Copy Detection.

HMM%2DBased Audio Keyword Generation.


Statistical modeling of complex backgrounds for foreground object detection.
This paper addresses the problem of background modeling for foreground object detection in complex environments. A Bayesian framework that incorporates spectral, spatial, and temporal features to characterize the background appearance is proposed. Under this framework, the background is represented by the most significant and frequent features, i.e., the principal features, at each pixel. A Bayes decision rule is derived for background and foreground classification based on the statistics of principal features. Principal feature representation for both the static and dynamic background pixels is investigated. A novel learning method is proposed to adapt to both gradual and sudden &#034;once-off&#034; background changes. The convergence of the learning process is analyzed and a formula to select a proper learning rate is derived. Under the proposed framework, a novel algorithm for detecting foreground objects from complex environments is then established. It consists of change detection, change classification, foreground segmentation, and background maintenance. Experiments were conducted on image sequences containing targets of interest in a variety of environments, e.g., offices, public buildings, subway stations, campuses, parking lots, airports, and sidewalks. Good results of foreground detection were obtained. Quantitative evaluation and comparison with the existing method show that the proposed method provides much improved results.
Visualization and User%2DModeling for Browsing Personal Photo Libraries.
We present a user-centric system for visualization and layout for content-based image retrieval. Image features (visual and/or semantic) are used to display retrievals as thumbnails in a 2-D spatial layout or &#034;configuration&#034; which conveys all pair-wise mutual similarities. A graphical optimization technique is used to provide maximally uncluttered and informative layouts. Moreover, a novel subspace feature weighting technique can be used to modify 2-D layouts in a variety of context-dependent ways. An efficient computational technique for subspace weighting and re-estimation leads to a simple user-modeling framework whereby the system can learn to display query results based on layout examples (or relevance feedback) as provided by the user. The resulting retrieval, browsing and visualization engine can adapt to the users&#039; (time-varying) notions of content, context and preferences in presentation style and interactive navigation. Monte Carlo simulations with machine-generated layouts as well as pilot user studies have demonstrated the ability of this framework to model or &#034;mimic&#034; users, by automatically generating layouts according to their preferences.
Trajectory%2Dbased ball detection and tracking with applications to semantic analysis of broadcast soccer video.
This paper first presents an improved trajectory-based algorithm for automatically detecting and tracking the ball in broadcast soccer video. Unlike the object-based algorithms, our algorithm does not evaluate whether a sole object is a ball. Instead, it evaluates whether a candidate trajectory, which is generated from the candidate feature image by a candidate verification procedure based on Kalman filter, is a ball trajectory. Secondly, a new approach for automatically analyzing broadcast soccer video is proposed, which is based on the ball trajectory. The algorithms in this approach not only improve play-break analysis and high-level semantic event detection, but also detect the basic actions and analyze team ball possession, which may not be analyzed based only on the low-level feature. Moreover, experimental results show that our ball detection and tracking algorithm can achieve above 96 % accuracy for the video segments with the soccer field. Compared with the existing methods, a higher accuracy is achieved on goal detection and play-break segmentation. To the best of our knowledge, we present the first solution in detecting the basic actions such as touching and passing, and analyzing the team ball possession in broadcast soccer video.

Nonparametric color characterization using mean shift.

A mid%2Dlevel representation framework for semantic sports video analysis.
Sports video has been widely studied due to its tremendous commercial potentials. Despite encouraging results from various specific sports games, it is almost impossible to extend a system for a new sports game because they usually employ different sets of low-level features appropriate for the specific games and closely coupled with the use of game specific rules to detect events or highlights. There is a lack of internal representation and structure to be generic and applicable for many different sports. In this paper, we present a generic mid-level representation framework for semantic sports video analysis. The mid-level representation layer is introduced between the low-level audiovisual processing and high-level semantic analysis. It allows us to separate sports specific knowledge and rules from the low-level and mid-level feature extraction. This makes sports video analysis more efficient, effective, and less ad-hoc for various types of sports. To achieve robustness of the low-level feature analysis, a non-parametric clustering, mean shift procedure, has been successfully applied to both color and motion analysis. The proposed framework has been tested for five field-ball type sports covering duration of about 8 hours. Experiments have shown its robust performance in semantic analysis and event detection. We believe that the proposed mid-level representation framework can be used for event detection, highlight extraction, summarization and personalization of many types of sports video. Categories and Subject Descriptors H.3.1 [Content Analysis and Indexing]: abstracting methods, indexing methods.
Robust moving video object segmentation in the MPEG compressed domain.

A semi%2Dfragile object based video authentication system.
This paper presents a semi-fragile object-based authentication solution for MPEG4 video. To protect the integrity of the video objects / sequences, a content-based watermark is embedded into each frame in the Discrete Fourier Transform (DFT) domain before the MPEG4 encoding. A set of Angular Radial Transformation (ART) coefficients are selected as the robust features of the video objects. Error Correction Coding (ECC) is employed for watermark generation and embedding. Using this methodology, our semi-fragile authentication solution can robustly tolerate some natural object manipulations and errors (e.g. translation, scaling, rotation, lossy compression, segmentation errors, etc) while securely preventing other malicious modifications. Experimental results further demonstrate that the proposed method can achieve a good tradeoff among system robustness, security and complexity. 1.
Content%2DBased Summarization for Personal Image Library.

Semantic Shot Classification in Sports Video.
In this paper, we present a unified framework for semantic shot classification in sports videos. Unlike previous approaches, which focus on clustering by aggregating shots with similar low-level features, the proposed scheme makes use of domain knowledge of specific sport to perform a top-down video shot classification, including identification of video shot classes for each sport, and supervised learning and classification of given sports video with low-level and middle-level features extracted from the sports video. It is observed that for each sport we can predefine a small number of semantic shot classes, 5~10, which covers 90~95 % of sports broadcasting video. With supervised learning method, we can map the low-level features to middle-level semantic video shot attributes such as dominant object motion (a player), camera motion patterns, and court shape, etc. On the basis of the appropriate fusion of those middle-level shot classes, we classify video shots into the predefined video shot classes, each of which has a clear semantic meaning. The proposed method has been tested over 4 types of sports videos: tennis, basketball, volleyball and soccer. Good classification accuracy 85~95 % has been achieved. With correctly classified sports video shots further structural and temporal analysis will be greatly facilitated.
Home Photo Content Modeling for Personalized Event%2DBased Retrieval.
this article, we approximate event by visual event, defined as an event based on the visual content of photos (that is, the &#034;what&#034; aspect)
Home Photo Content Modeling for Personalized Event%2DBased Retrieval.
this article, we approximate event by visual event, defined as an event based on the visual content of photos (that is, the &#034;what&#034; aspect)
Evaluation of salient point techniques.
In image retrieval, global features related to color or texture  are commonly used to describe the image content. The probleln with  this approach is that these global features cannot capture all parts of  the image having different characteristics. Therefore, local computation  of image information is necessary. By using salient points to represent  local information, more discriminative features can be computed. In this  paper we compare a wavelet-based salient point extraction algorithm  with two corner detectors using the criteria: repeatability rate and information  content. We also show that extracting color and texture information  in the locations given by onr salient points provides significantly  improved results in terms of retrieval accuracy, computational complexity,  and storage space of feature vectors as compared to global feature  approaches.
A unified framework for semantic shot classification in sports videos.
In this demonstration, we present a unified framework for semantic shot classification in sports videos. Unlike previous approaches, which focus on clustering by aggregating shots with similar low-level features, the proposed scheme makes use of domain knowledge of specific sport to perform a top-down video shot classification. That is, combining with inherent game rules and television field production, for each sport through careful observations we predefine a set of semantic shots which cover 90 to 95 % of sports broadcasting video. Under the supervision of predefined shots set, we map the low-level features to high-level semantic video shot attributes such as dominant object motion (a player), persistent camera panning, and court shape. On the basis of the appropriate fusion of those high-level shot attributes, we classify video shots into several predefined categories, each of which has a clear semantic meaning. The experiments show that, compared to traditional clustering methods and key-frame based analysis, the proposed framework features great capability of semantics mining. Due to remarkable structure constraints and limited sports photography, this framework provides a generic solution for sports video shot classification, which can be adapted to a new sport type without major modification. With correctly classified sports video shots further structural and temporal analysis will be greatly facilitated.
Clear face analysis from MPEG compressed video.

A robust and secure media signature scheme for JPEG images.
Abstract — In [1, 2, 3], we have briefly introduced a robust and secure digital signature solution for multimedia content authentication, by integrating content feature extraction, error correction coding (ECC), watermarking and cryptographic signature into one unified framework. In this paper, we firstly study the difference between two types of distortions, namely incidental distortions which come from the defined acceptable content manipulations such as lossy compression and intentional distortions which come from real attacks such as content copy-paste. We then introduce a new JPEG-compliant solution under our proposed framework but with different ECC and watermarking methods. Finally system performance evaluation and analysis will be given to further demonstrate the practicability of our method.
Evaluation of Salient Point Techniques.
In image retrieval, global features related to color or texture  are commonly used to describe the image content. The probleln with  this approach is that these global features cannot capture all parts of  the image having different characteristics. Therefore, local computation  of image information is necessary. By using salient points to represent  local information, more discriminative features can be computed. In this  paper we compare a wavelet-based salient point extraction algorithm  with two corner detectors using the criteria: repeatability rate and information  content. We also show that extracting color and texture information  in the locations given by onr salient points provides significantly  improved results in terms of retrieval accuracy, computational complexity,  and storage space of feature vectors as compared to global feature  approaches.
Visualization

Shot Classification of Sports Video Based on Features in Motion Vector Field.

Statistical Analysis of Musical Instruments.

Foreground Segmentation Using Motion Vectors in Sports Video.

Support Vector Machine Learning for Music Discrimination.

Similarity matching of continuous melody contours for humming querying of melody databases.
Music query-by-humming is a challenging problem since the humming query inevitably contains much variation and inaccuracy. Many of the previous methods, which adopt note segmentation and string matching with dynamic programming, suffer drastically from the errors in the note segmentation which affects retrieval accuracy and efficiency. In this paper, we present a novel melody similarity matching technique which is based on continuous melody contour. The continuous melody contour has many advantages over previous string based melody representation. We introduce a contour alignment technique which addresses the robustness and efficiency issues. We also present a new melody similarity metric which is performed directly on continuous melody contours of the query data. This approach cleanly separates the alignment and similarity measurement in the retrieval process. We show how to robustly and efficiently align the query melody with the reference melodies and how to measure the similarity subsequently. Our melody alignment method can reduce the matching candidate to 1.7% with 90% correct alignment rate. The overall retrieval system achieved 80% correct retrieval in the top 10 rank list.
A robust and secure media signature scheme for JPEG images.
Abstract — In [1, 2, 3], we have briefly introduced a robust and secure digital signature solution for multimedia content authentication, by integrating content feature extraction, error correction coding (ECC), watermarking and cryptographic signature into one unified framework. In this paper, we firstly study the difference between two types of distortions, namely incidental distortions which come from the defined acceptable content manipulations such as lossy compression and intentional distortions which come from real attacks such as content copy-paste. We then introduce a new JPEG-compliant solution under our proposed framework but with different ECC and watermarking methods. Finally system performance evaluation and analysis will be given to further demonstrate the practicability of our method.
Foreground Object Detection in Changing Background Based on Color Co%2DOccurrence Statistics.

Spatial Visualization For Content%2DBased Image Retrieval.
In traditional content-based image retrieval (CBIR), the retrieved images are displayed in order of decreasing similarities from the query and can be considered as a 1-D display. In this paper a novel optimized technique is proposed to visualize the retrieved images not only in order of their decreasing similarities but also according to their mutual similarities visualized on a 2-D screen. Principle Component Analysis (PCA) is first performed on the retrieved images to project the images from the original high dimensional feature space to 2-D screen. The result of PCA analysis is denoted as a PCA Splat. To minimize the overlap between images, a constrained nonlinear optimization approach is used. The experimental results show a more perceptually intuitive and informative visualization of the retrieval results. The proposed technique not only provides a better understanding of the query results but also aids the user in forming a new query.
Display Optimization for Image Browsing.
In this paper, we propose a technique to visualize a multitude of images on a 2-D screen based on their visual features (color, texture, structure, etc.). The resulting layout will automatically display the mutual similarities of the viewed images. Furthermore, audio features, semantic features, or any combination of the above can be used in such a visualization. The original high dimensional feature space is projected on the 2-D screen based on Principal Component Analysis (PCA). PCA has the desired property of being simple, fast and unique (i.e. repeatable) and the only linear transformation that achieves maximum distance preservation in projecting to lower dimensions. Furthermore, we have developed a novel technique for solving the problem of overlapping (obscured) images shown in the proposed 2-D display. Given the original PCA-based visualization, a constrained nonlinear optimization strategy is used to adjust the position and size of the images in order to minimize overlap (maximize visibility) while maintaining fidelity to the original positions which are indicative of mutual similarities. A significantly improved visualization of large image sets is achieved when the proposed technique is applied.
Content%2Dbased image retrieval using wavelet%2Dbased salient points.
Content-based Image Retrieval (CBIR) has become one of the most active research areas in the past few years. Most of the attention from the research has been focused on indexing techniques based on global feature distributions. However, these global distributions have limited discriminating power because they are unable to capture local image information. Applying global Gabor texture features greatly improve the retrieval accuracy. But they are computationally complex. In this paper, we present a wavelet-based salient point extraction algorithm. We show that extracting the color and texture information in the locations given by these points provides significantly improved results in terms of retrieval accuracy, computational complexity and storage space of feature vectors as compared to the global feature approaches.
Image retrieval using wavelet%2Dbased salient points.
Content-based Image Retrieval (CBIR) has become one of the most active research areas in the past few years. Most of the attention from the research has been focused on indexing techniques based on global feature distributions. However, these global distributions have limited discriminating power because they are unable to capture local image information. Applying global Gabor texture features greatly improve the retrieval accuracy. But they are computationally complex. In this paper, we present a wavelet-based salient point extraction algorithm. We show that extracting the color and texture information in the locations given by these points provides significantly improved results in terms of retrieval accuracy, computational complexity and storage space of feature vectors as compared to the global feature approaches.
Discriminant%2DEM Algorithm with Application to Image Retrieval.
In many vision applications, the practice of supervised learning faces several difficulties, one of which is that insufficient labeled training data result in poor generalization. In image retrieval, we have very few labeled images from query and relevance feedback so that it is hard to automatically weight image features and select similarity metrics for image classification. This paper investigates the possibility of including an unlabeled data set to make up the insufficiency of labeled data. Different from most current research in image retrieval, the proposed approach tries to cast image retrieval as a transductive learning problem, in which the generalization of an image classifier is only defined on a set of images such as the given image database. Formulating this transductive problem in a probabilistic framework, the proposed algorithm, Discriminant-EM (D-EM), not only estimates the parameters of a generative model, but also finds a linear transformation to relax the assumption of pro...
Combine User Defined Region%2Dof%2DInterest and Spatial Layout for Image Retrieval.
Content-Based Image Retrieval (CBIR) is one of the most active research areas in recent years. Many visual feature representations have been explored and many systems built. However, in most of current systems, only the global features such as overall color histogram and texture moments are used which ignore the actual composition of the image in terms of internal objects. Although relevance feedback was proposed [1] to incrementally supply more information, they may fail due to the lack of higher-level information about what exactly was of interest. Since automatic segmentation of Regionof -Interest (ROI) is not always reliable, human assistance is necessary. In this paper, a novel approach combining user defined Region-of-Interest and spatial layout is proposed for CBIR. Better capture of image object is achieved by the user rather than the computer. Therefore, more accurate relevance feedback is achieved and thus leads to a more powerful search engine.
Incorporate Support Vector Machines to Content%2DBased Image Retrieval with Relevant Feedback.
By using relevance feedback [6], Content-Based Image Retrieval (CBIR) allows the user to retrieve images interactively. The user can select the most relevant images and provide a weight of preference for each relevant image. The high level concept borne by the user and perception subjectivity of the user can be captured by the system to some degree. This paper proposes an approach to utilize both positive and negative feedbacks for image retrieval. Support Vector Machines (SVM) is applied to classifying the positive and negative images. The SVM learning results are used to update the preference weights for the relevant images. This approach releases the user from manually providing preference weight for each positive example. Experimental results show that the proposed approach has improvement over the previous approach [5] that uses positive examples only.  1. 
Integrating Unlabeled Images for Image Retrieval Based on Relevance Feedback.
Retrieval techniques based on pure similarity metrics are often suffered from the scales of image features. An alternative approach is to learn a mapping based on queries and relevance feedback by supervised learning. However, the learning is plagued by the insufficiency of labeled training images. Different from most current research in image retrieval, this paper investigates the possibility of taking advantage of unlabeled images in the given image database to make feasible a hybrid statistical learning. Assuming a generative model of the database, the proposed approach casts image retrieval as a transductive learning problem in a probabilistic framework. Our experiments show that the proposed approach has a satisfactory performance in image retrieval applications.  1. Introduction  To avoid manual annotating large image databases, an alternative approach of retrieving images is content-based image retrieval (CBIR), by which images would be indexed by their visual contents such as c...
Incorporate Discriminant Analysis with EM Algorithm in Image Retrieval.
One of the difficulties of Content-Based Image Retrieval (CBIR) is the gap between high-level concepts and low-level image features, e.g., color and texture. Relevance feedback was proposed [1] to take into account of the above characteristics in CBIR. Although relevance feedback incrementally supplies more information for fine retrieval, two challenges exist: (1) the labeled images from the relevance feedback are still very limited compared to the large unlabeled images in the image database. (2) relevance feedback does not offer a specific technique to automatically weight the low-level feature. In this paper, image retrieval is formulated as a transductive learning problem by combining unlabeled images in supervised learning to achieve better classification. Experimental results show that the proposed approach has a satisfactory performance for image retrieval applications.
Update Relevant Image Weights for Content%2DBased Image Retrieval using Support Vector Machines.
Relevance feedback [1] has been a powerful tool for interactive Content-Based Image Retrieval (CBIR). During the retrieval process, the user selects the most relevant images and provides a weight of preference for each relevant image. User&#039;s high level query and perception subjectivity can be captured to some extent by dynamically updated low-level feature weights based on the user&#039;s feedback. However, in MARS [2] only the positive feedbacks, i.e., relevant images are considered. In this paper, a novel approach is proposed by providing both positive and negative feedbacks for Support Vector Machines (SVM) learning. The SVM learning results are used to update the weights of preference for relevant images. Priorities are given to the positive feedbacks that have larger distances to the hyperplane determined by the support vectors. This approach releases the user from manually providing preference weight for each positive example, i.e., relevant image as before. Experimental results show ...
Finging Near%2DOptimal Build Orientations for Shape Deposition Manufacturing.

Digital Video Analysis and Recognition for Content%2DBased Access.

